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*> The Instructional Effectiveness of Random* Logical and 
Ordering Theory Generated Learning Hierarchies 

Models of Instructional development generally Include one or more 
steps for analyzing the terminal objective Into a sequence of prerequi- 
site objectives (Glaser^ 196S; Brlggs» 1970)* To meet the need for a 
technology of task analysis > Gagne (1962) suggested a research methodo- 
logy which would yield hierarchies of learning taska. The Gagn^ model 
and Its subsequent modification (Gagne» l968;.Whlte» 1973» 1974b) 
require teaching the elements of the proposed hierarchy to a group 
of subjects to empirically validate the proposed hierarchy* 

Proponents of an alternative method of hierarchy generation have 
advocated schemes which oinlt' the controlled Instruction as a part of the 
empirical validation (Phillips and Kane, 1973; Alraslan^ 1971; Alrasl^n 
and Bart, 1973) « By assessing the performance patterns of a group with 
^^Isperslon of scores on the objectives^ a hierarchy or network of 
objectives Is derived. It*s supporters have suggested that psychometric- 
ally generated hierarchies and sequences may be a favorable alternative 
to the time consuming and expensive task of the Gagne model. 
Alraslan and Bart (Alraslan^ 1971;^ Alraslan and Bart» 1971; Bart and 
Krus, 1973) have proposed the ordering theory model as a psychometric 
means of Identifying networks of relationships among skills. 

Whereas, the Gagne hierarchy generation procedure Is based upon s ' 
logical task analysis of the terminal objective, ordering theory psycho* 
metrically compares all possible prerequisite relationships which exist - 
In the total set of objectives. By examining tha patterns of test respon* 
ses on the set of objectives, the ordering theory analysis produces a 
network of hierarchy among the objectives* 
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The notation used In ordering theory analysis Is similar to that of 
the Gagne hierarchical connection. In a two Item hierarchical connection^ 
X and y» there are four possible response patterns: (0»0)» (1|0); (!»!)» 



and (0^1) • Bart and Alraslan propose that the first three response pat- 
terns are confirmatory and that the last (0»1) Is dlsconf Irmatory of the 
hierarchical relationship* The absence of dlsconf Irmator^ response 
patterns validates the hypothesized ordering (Alraslan and Bart» 1973). 

Ordering theory Is a deterministic model as opposed to most Inferen- 
tlal statistical models which are probabilistic. Although It Is derived 
from Scalogram analysis^ ordering theory permits the testing of non- 
linear task hierarchies. It may be used to confirm the logical existence 
of an a priori hierarchy » or In those cases where no a priori hierarchy 
has been posited^ to suggest tfje £est fitting hierarchical relationships 
of a set of tasks (Alraslan an| Bart» 1971). 

Although several researchers haveadvocated use of the ordering theory 

^ ; 

model for hierarchy generation!^ there Is an absence of empirical evidence 



bti the Instructional effectiveness of sequences produced by the ordering 
theory analysis. The primary purpose of this study was to empirically 
test the validity of a hierarchy generated from tha ordering theory pro- 
cedures. Specif legally does the sequence generated from an ordering theory 
analysis result In greater Instructional gains than a sequence generated 
from a logical Gagne task analysis? Secondly » will the ordering theory 
hierarchy prove to be valid as empirically assessed by the White and 
Clark test of Inclusion (1973)? The test of Inclusion provides for an 
estimate of the error of measurement In testing the hypothesis that **all 
the members of a population who possess a certain skill are a subset of 
the members who possess another skill** (Whlt^ and Clarke 1973» p. 77). 



\ 
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In evaltiatlng the usefulness of any IndtiMctlonal procedure a neces* 
saryi but not sufficient criterion for acceptance Is Its ability to 
product Itarnlng. However, It Is possible for two approaches to teaching 
a skill to both result In mastery of that skill, but requiring unequal 
amounts of time to do so* Thus, all other factors being equal, the 
Instructional procedure requiring less time would be more efficient* ,In 
comparing the experimental sequences In this study, time required for 
subjects to complete the Instructional program was also Included as a •> 
dependent measure. 
Generation of Learning Hierarchies 

Although all subjects In the main experiment received Instructional 
programs containing the ^ame, frames and test Items their order varied 
according to the lnst:ructlonal Sequence; 1) a logical seqtie^ce generated 
by Gagn^ type task analysis, 2) a sequence derived from ordering theory 
analysis and 3) sequences Individually randoml'^ed* 

A task analysis of the terminal objective *'Glven the mean and 
standard deviation of a distribution, to calculate the percentage of cases 
that fall below a specified score** by the author with recommendations of 
two professors of education who had previously taught statistics or measure 
ments courses. By successively asking the question **What would the 
Individual already have to know how to do In order to learn this new cap^ 
ability simply by being' given verbal Instructions?** (GagiTe, 1962, p. 357), 
the hierarchy depicted In Figure 1 was derived* The Instructional objec* 
'tlves corresponding to the numerals are listed In Table 1. 

INSERT FIGURE 1 APPROXIMATELY. HERE 

1 

r^SERT TABLE 1 APPROXIMATELY HERE 
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To constri^ct an ordering theory hierarchy a matrix of the percentage ^ 
of dlsconf Irmator^Kresponse pattern^ (0»1) for all pair combinations of 
objectives was developed (see Table 2). The set of test questions de'slg- 
nated as the posttest for the main experiment was used In the construction 
of the ordering theory hierarchy. Thc^ criterion level for the ordering 
theory hierarchy was -lO, which meant that a connection between two 
objectives was determined to be hierarchical If less than lOZ of the total 
response patterns were -dlsconf Irmatory (0,1). For example^ In Table 2^ 
element or objective 1 Is determined to be prerequisite ^o objective 2» as 
only 6% of the response patterns were dlsconf Irmatory (I.e., only six per-* 
cent of the students who missed objective 1 were able to pass objective 2). 



INSERT TABLE 2 APPROXIMATELY HERE 



There were marked differences between the hierarchical structures 
generated by the two procedures (see Figures 1 and 2). A number of postu* 
lated connections were reversed by the ordering theory analysis, such as 
the 11/12 connection. The most notable occurrance seems to be the IsoJ^atlbn 
of objective 16 In the ordering theory hierarchy. Additionally, the complex 
network generated by ordering theory analysis suggests the possibility of 
slternate learning paths to higher objectives. 

After the learning hierarchy had been proposed by the Gagne task 
analysis^ a pool of nine test Items per objective was developed. These 
were randomly divided Into three sepa^te tests each containing three test 
Items for each of the seventeen Instructional objectives. The three tests 
were Identified as pretest, posttest and retention test for the main experi- 
ment. To obtain a sample which had had previous exposure to the terminal 



and enabling objectives^ S4 advanced undergraduate and graduate students 
received one of the three alternate test forms. Sixty percent of the 
colXftga sample were selected at random to receive the i^lternate form used 
to formulate the ordering theory hierarchy* The test Items were arranged 
Into a test booklet such that Items 'measurln3'objectlves highest In the 
proposed hierarchy appeared first and the Items from the lowest level of 
the hierarchy appeared J.ast to minimize the effects of learning from .the 
test. Performance on each objective was scored pass or fall, with two 
Items out of three answered correctly being scored as a paSS* One way 
analysis of variance showed no significant (p.<OS) differences In the 
number of objectives mastered among the three groMps of the college sample* 

The program frames were adapted from the programmed text Statistical 
Concept ! A Ba sic Program by Amos^ Brown, and Mink (1965). Permission to 
use the text was obtained from the publishers, Harper and Row and Company. 
The authors (Amos^ Brown^ and Mink, 196S) reported five separate validation 
and error rate studies using the program with error ratee of five to seven 
percent and significant Instructional gains* 

In proposlng^a refinement In the Gagne model of hierarchy validation, 
WhJte (1973) suggested that 1) posttest items for each objective appear 
Immediately after Instruction for that objective, and 2) that two or more 
questions per objective be used to permit. the estlouite of the error of 
measurement. To minimize the effects of error of measurement Bart and 
Alraslan (1974) have suggested the use of open-end^d questions* Each of 
^ these recomniendatlons was followed with the use of three Items P^r objec- 
tive In the pretest^ embedded posttest^ and the retention test* The , ^ 
embedded posttest served twa functions: 1) to provide a gain score on which 
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the Instructional effectiveness o£ the three sequences could be compared, 
and 2)' to provide test performance patterns £or a hierarchy validation 
using the White and Clark test o£ Inclusion (1973). Although each set o£ 
programmed booklets contained the same frames and same test Items, their 
order varied according to the Instructional sequence. 



INSERT FIGURE 2 APPROXIMATELY HERE 
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METHOD 

Subjects 

The sample £or the main experiment consisted o£ 57 high students £rom 
two Introductory psychplogy classes. Forty^seven were seniors and ten 
were juniors. In addition, a quasl*-control group .o£ 12 seniors was given 
the pretest and the retention test to obtain additional lu£ormatlon on the 
Instructional effectiveness o£ the program. 

Materials - 
Each subject In the main experiment received a packet o£ 6-1/2 by 11 
'^nch (22cm x 28cm) mimeographed pi^ogrammed booklets^ Each o£ the 17-objec* 
tlves was contained In a separate booklet. The posttest Items appeared 
on the last page o£ each booklet. Each booklet was collected before the 
subject could begin Instructlon^on the next objective In his sequefice. 
The booklets were se<]uentlally numbered as dictated by the experimental 
condition. Each booklet In the packet also contained the, subject 's 
Identification number. 
Procedures 

All subjects In the main experiment were given a pratest on all 

8 
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enabling objectives and the terminal objective. The pretest ^tenis were 
mindomly ordered^ to avoid possible contaminating effects of the pretest 
aequence serving as an advance organizer. 

Five days after the pretest the programined booklets were glven^to 
all of the high school subjects. They responded In the booklets to each 
frame and to all posttest Items.. After a subject made ovett response 
to each frame within the Instructional program^ he uncovered the correct 
answer. 

All posttest questions Immediately followed the corresponding* 
Instruction within -the programmed text. Thus, each student was required 
to respond to the test Items relating to a particular objective before 
proceeding to the Instruction for the next objective In the sequence. A 
retention test was glven*all three groups approximately five daya after 
all students bad complet^ed the Instructional program. 

Each subject^s performance on each of the objectives w^s obtained 
from the test Items embedded within the program^ following the recommenda* 
tlons of White (1973). A score of paas or fall was recqrded for each 
objective^ with two Items out of three answered tiorrectly being scored as 
a pass. The same criterion level was used for the pretest and the retention 
test. 

Because the test Items composed a criterion-referenced Instruoent^ the 
traditional nom^referenced measures of reliability were Inadequate and In* 
appropriate as the deviation from the mean would be misleading. A criterion- 
referenced correlation ccef flclei:\t developed by Livingston (1970) waa used 
for this purpose. This procedure produced a crlterlon*-referenced correlation 
of .932. 
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RESULTS 



Covar lance s on Post and Retention Test Scores 

As rtconuiended by White (1974) the Individual posttestd for each 
objective were £lven to all subjects Immediately after completion of the 
Instruction on that objective and before the students proceeded to the next 
objective In the sequence. If a student answered correctly two of tht 
three questions for eact} objective he was credited with a pass for that 
objective. 

An analysis of covarlance on the number of objectives passed on the 
posttest was computed with the pretest as the covarlatr. No significant 
differences were found among the three groups (p > .05). .Means and standard 
deviations oh' the pretest and posttest for the three experimental conditions 
are 'depicted In Table 3. 

INSERT TABLE 3 APPROXIMATELY HERE 

INSERT fABLE 4 APPROXIMATELY HERE 
V ^ 

As shown In Table 4 no significant differences were found among the 
three treatment groups In posctest performance when tht pretest was held 
as covarlate. No significant differences (p > .05) were found from entlysls 
of covarlance on objectives passed on the retention test with the posttti^t 
as covarlate. (See Table 5). 



INSERT TABLE 5 APPROXIMATELY HERE 



Comparisons of Time to Completion 

The total number of minutes that each student spent, working on the 
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program and embedded posttpsts was recorded. The Qeans and standard 

it 

deviations are shovm In Table 6* 

The nost notable occurrance Is, of course, the large difference In 
average number of minutes to complete the Instructional program between 
the random sequence group (X 123.21) and the two ordered sequences 
(X ordering theoretic * 104.05 and X logical 106.263). The analysis of 
variance produced an F-ratlo of S.792 which was significant at the .01 
level (see Table ?). Post hoc contrasts with the Scheffe method yielded 
ranges of 4.48 and 4.48 for both contrasts against the random sequence. 
Thus, both logical and ordering theory sequences differed significantly 
(p < .01) frqm the random sequence, but did not difffer significantly^ from 
each other. 



INSERT TABLE 6' APPROXIMATELY HERE 
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Hierarchy Validations 

For eac^ proposed hierarchical conniictlon In both the ordering theory 
'and logical sequences a 4 x 4 table of performance combinations was con^ 
structed. An example of such a contingency table Is shown In Figure 3. 



INSERT FIGURE 3 APPROXIMATELY HERE 



By contracting the observed and expected frequencies In the major 
dlsconflrmatory cell (0,3 In the above matrix) a critical value for 
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defining a hierarchical relationship Is derived. The critical value Is the 
maximum u£miber of cases which may occur In the 0,3 cell co accept the 
proposed hierarchical ccnnectlon. Under the null hypothesis that critical 
value Is the number which ndght achieve a 0,3 response pattern by errors 
of measurement alone. Llnke (1974) suggests a more tolerable level than 
allowed by error of measurement alone to penult ^legitimate exceptions^ 
arising through the use of unldentlf le(^ prerequisite^ skllls^or alternate 

c 

learning pathways" (p. 915). This Is In agreement with Gagne*s (1970) 
proposal for substantial rather than absolute levels of acceptance. Be- 
cause the number of cases constituting each hierarchy sample was relatively 
small (19) the null hypothesis was tested at the .01 level as well as the 
.00 level. Acceptance of a connection as hierarchical un^er the .00 null 
hypothesis limits the critical value to only those dlsconf Irmatory cases 
which could be expected to occur through errors of measurement alone. The, 
summary of results of the validations^ Is presented In Tables 8 and 9. As 
an example from Table 9 the Gagne analysis sujfgested that objective jl was 
a prerequisite to objective #2. The White and Clark' (1973) test of Inclusion 
permitted only one dlsconf Irmatory to occur as a result of error of measure- 
ment^under the .01 hypothesis, which Is always more ret^trlctlv^ than the 
.00 null hypothesis. As two cases occurred which were dlsconf Irmatory, the 
propised hierarchical connection was rejected. 



INSERT TABLE 6 APPROXIMATLEY HERE 



INSERT TABLE 9 APPROXIMATELY HERE 



Of the\^4 postulated hierarchical connections In the ordering theory 
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network, 21 were accepted as valid hierarchical connections. Fourteen of 
th« nineteen postulated connections In the logical hierarchy were accepted 
at the *0l hypothesis level* 
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DISCUSSION 

Many of today^d educationaldevelopmentSi including competency -based 
education, auto -instructional programs, and mastery models of learning* 
are based upon the assumption that an optimal sequence for successful 
instruction exists. The present study seems to partially support that 
assumption. Although no significant differences were found between the 
logical, ordering theory and random sequences* the random sequence, group 
did take significantly (p < .01) longer to complete the instruc :ional 
program. Thus, subjects exposed to a random sequence of objectives were 
able to learn, but it took them approximately 17Z longer to do so. If time 
were constant for all three groups* it Is probable that the number of 
objectives mastered would have been significantly less for the random 
sequence subjects than for the two ordered sequence groups. 

Roe, Case and Roe (1961) failed to find significant differences 
between the posttedt scores of logical and random sequence presentations 
with learning programs which took apprpximately 45 minutes for completion. 
Obtaining similar results, Hamilton (1964]^ compared logical and random 
sequences of an instructional program which required about' 24 minutes 
to complete. In a later review Tennyson (1972) suggested that the brevity 
of th6 programs might have permitted short term memory to compensate for 
ordering effects. The instructional program of the present study required 
a mean time of 111.18 minutes to complete. Although this seems signifi- 
cantly longer than the programs used, by Roe a|id Hamilton, relatively high 
pretest scores (X 6. 526) could sufficiently reduce;tha amount of informs- 
tion which vould need to be ordered by a subject given a random sequence. 

Although posttesting for each skill immediately after its instruction 
would seem to limit the effects for short term memory, it is difficult to 

14 



be conclusive as the skills were arranged In a multi-branched non-linear 
hierarchy* Hbre Important than the total number of objectives passed Is 
the response pattern of those objectives passed. This 1& confounded 
further by the pretest mastery of some parts of the hierarchy* The oaly 
statistical procedure which attempts to analyze the' 'Hon-llnear response 
patterns of a hierarchy Is the ordering theory analysis » which lacks the 
necessary estimate of error of measurement. Further^ the ordering theory 
procedure does not have a provision for comparing several different 
response patterns. It seems likely that more sophisticated techniques for 
analysis of non*llnear hierarchies oust be developed before the problem 
can bjB fully resolved. 

An additional explanation for the difficulty In. finding the effects of 
sequence order may be the size of the unit of task analysis. This question 
has been raised by Resnlck (1974). In examining Individual differences In 
progressing throut^h 1ilerarchles» she noted that many students seem able to 
learn several objectives or skills simultaneously. It may be that the 
trend has been toward an overly minute analysis. 

Perhaps Intellectual skills exist In clusters with dimensions which 
share vertical or horizontal transfer properties. Graphically these skills 
may be related In a Gagne^type' learning, hierarchy. The difference Is In 
how students progress through the hierarchy. It may be that higher order 
learning skills or abilities provide a third dimension to the two dimension* 
al learning hierarchy. A better understanding of what G^$ne\erms "cog-^ 
nltlve strategies" may provide a clua to the comprehension of Individual 
differences In learning the same sequence of Intellectual skills. Certain 
'Cognitive strategies may permit the learner to skip levels of the hierarchy 
or ^zondense them. The nature of the learning task must also be considered 
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as a possible variable In determining optimal sequences. The statistical 
skills developed through the learning program were primarily at the prin- 
ciple level of Gagne*s levels of learning (1970). Lacking evidence to 
the contrary It seems reasonable to believe that all skills are not equal 
In Importance or transfer value. In any hierarchy there may exist a few 
very key or crucial skills whose ""mastery will greatly enhance the learning 
of many other skills. ' 

Learning hierarchy researchers (Resnlck^ 1974;. and Phillips » 1974) 
have suggested the possibility of alternate learning hlet;archles. The 
possibility that a skill might be mastered by two or more routes Indicates 
that If a ''true hler,archy'* of all possible routes were known» a number of 
disjunctive connections would exist. A disjunctive connection leading to 
a higher element In a hierarchy would Indicate tyo or^more alternate skills^ 
any one of which Is sufficient to Insure mastery of the upper element. Ttadl 
tlonal forms of hierarchy generation allow for conjunctive connections 
where two or more Independent skills are jointly necessary for mastery 
of the higher skill. The statistical possibility of creating disjunctive 
connections Is established through the ordering theory analysis, but the 
empirical validation of such alternate pathways Is lacking. Further Investl*- 
gatlon III thl^rea might account for some of the Individual differences 
In learning Intellectual skills. 

If any of the theoretic possibilities described above are valld» the 
expectation of absolute hierarchies Is unrealistic;. Interactions with 
task content, l:earner characteristics such as cognitive strategies or 
affective elements may predetermine^ Individual differences In the ability 
to progress through any proposed learning hierarchy « 

An Inherent difficulty In conducting research based upon logical 
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task analyses, such as that of Gagne, 1b that the proposed hierarchy 1b 
always dependent upon the Intellectual skills and subject matter knowledge 
of one or a few Individuals. The possibility of errant analyBeB through 
omission of necessary stepB, Inclusion of extraneous Bkllls, or misalign- 
loent of elements within the hierarchy t must be considered. The more 
mlnlBcle the analysis, the greater the probability of committing such an 
error . 

Twenty-one of the 24 postulated connections for the ordering theory 
hierarchy were accepted as valid with the White and Clark (1973) teBt of 
IncluBlon. Of the nineteen connections In the logical hierarchy, four^ 
teen were accepted as valid. Although both hierarchies were Bubstantlally 
valid as defined by the teBt of Inclusion, the rejected connectlonB could 
account for the lack of significant dlfferenceB In mastery between the two 
ordered Bequences and the r^dom sequence. For exao^le, of the three 
postulated prerequisite sklllB^^for objective number IS In the logical 
hierarchy, two were not accepted as valid connectlonB. Ab objective IS 
waB In the middle of the hierarchy. It Is possible that Instruction on 
objectives 11, 1^, 13, and 14 was not necessary for success on objective - 
is. Barring the existence of unidentified prerequisites (which may.be a 
tenuous assumption), only objective or skill number 10 Is a necessary 
prerequisite. If ^ studeijt had passed objective 10 In (he pretest, or 
was able to mastet It through problem solving strategies^ he may master 
skills IS, 16, and 17 without receiving passing scores on objectives 11 
through 14 » giving him a low total score although passing higher level 
objectives. 

Another variable which must be considered Is the task level of the 
hierarchy elements. 'According to Gagne (^970) concepts are necessary for 
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the mastery of principles, and the acquisition of principles Is essential 
to successful problem solving. However, most of the tasks In this learn- 
ing situation were at the principle level. Had the ^hierarchy extended 

down to the concept level, or their prerequisites discriminations, the 

hlerarchlca^l nature of the learning tasks might be more pronounced, as 
supported by the findings of Gagne and Bassler (1963). Of course, a more 
naive sample must be used, as a mean of 6. 526 objectives were passed on 
the pretest by the high school subjects. 

The White and Clark (1973) test of Inclusion Is a relatively untested 
statistical procedure, as only two major studies (White, 1973; Llnke, 1974) 
have. employed the procedure to validate learning hierarchies. Two poten* 
tlal deficiencies mus^t be considered In using this technique In assessing 
the i?alldlty of hierarchical connections. 

The provision for an estimate of the £rror of measurement Is an 
attractive and valuable feature, but It mus^t be noted that the greater 
the error of measurement, the higher the critical value for acceptance ■ 
of the connection as hierarchical. Thus, less reliable tests are more 
likely to yield valid connections. 

A second reservation to be considered Is that If ths successes on 
the postulated superordlnate skill Is equal to or less than the 
critical value for thai connection. It Is statistically Impossible to 
reject the connectloln. This Is true be&ause the hierarchical connection 
Is rejrtcted only when the number of 0,1 (failure on subordinate task and 
mastery of the superordlnate task) response patterns exceed the critical 
value. The same situation exists whenever the number of subjects falling 
the subordinate task Is equal to or less than the critical value. In 
examining the difficulty posed by this occurrance^ White (1973) suggested 

18 



ERIC 



18 



that the hierarchy be verified by a subject matter expert to establish 
an Initial ^'logical validity*" 

The central assun^tlon underlying learning hierarchy research has 
been that failure to learn Is attributable primarily to a lack of pre* 
requisite skills* Although substantial research has established that 
sequence of skill Instruction does have a significant role (Gaghe^ 1962; 
Gagne and Bassler, 1963) It seems unwise to consider the learning of 
Intellectual skills only In terms of prerequisite skills* Interactions 
with other Yarlat>les must be examined* For leam;Lng hierarchies to 
have a profound effect in the classroom, more efficient procedures for 
task analysis or hierarchy generation are needed* Improvements upon 
the ordering theory analysis may provide promise In this direction* 
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Figure i .Learning hierarchy derived fron Gagne taatc^analysla 
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Table 1 

Tnst'ructlonal Objectives Arranged According 

To 

Position In The Logical Sequence 



li Given a set of scores to Identify the frec^uency of a 
specified QCore 

*i 

2. Given a group of numbers representing test scores^ 
to order according to magnitude In a distribution 

3* Given a set of numbers to arrange In frequency 
distribution 

4. Given a completed frequency polygon, to correctly 
Identify the frequency of a specific score 

5. Given a histogram, to Identify the .frequency with 
whlch^a specified score occurs 

6. Given a group of numbers to correctly construct a 
frequency polygon 

7. Given a group of nuad)ers to construct 'a frequency ^ 
histogram 

8. Given an odd number of scores, the student will 
correctly Identify the numbev which repr^eeeats the 
median 

9. Given a group of numbers to oelculate the meea 

10. Given three. graphs representing normal curves to select 
the one which represents the most variable group 

11* Given N and a score rank, to specify the percentile 
of the specified ^core 

12 i Given N and a percentile^' to specify the rank of e o 
specified score 
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Table 1 (Continued) 
Instructional Objectives Arranged According 

To 

Position In The Logical Sequence 



.13. Given any two Integers (^3 to H^3) designated as I 
standard deviations > to calculate the percentage , 
of cases falling between those two points in a 
normal distribution 

14. Given an integer between -3 and H^3 rrtpresentlng a 
standard deviation of a distribution of scores* to ^ 
specify the percentage of cas^s w|[ilch fall above and 
below that point 

15. Given the mean and standard deviation of a distribution 
of scores* tovcalculate the test score which would 
fall on any given standard deviation point ebove or 
below the mean 

16. Given a test score, the mean, and the standard 
^devlatlon^ to calculate the appropriate z^^score for 
that test score 

17. ^ Given the mean and standard ^deviation of a , 

distribution, to calculate the percentage of 

cases chat fall Q>elow a specified score (TERMINAL OBJECTIVE) 
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Table 2 

?erreatage of Olscoflf Itmatorjr R«spona* Patterns 
(M - 33). 



01 















Upper Elcnetit 




















I tea 
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2 


3 


4 
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6 
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9 


10 


11 


12 


13 


14 


15 


16 


17 
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27 


27 








15 


12 


9 


12 
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0 


0 


0 
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64 
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33 
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36 
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33 
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30 
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3 




3 
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A 
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0 
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A 
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, Table 3 

Pre£«st( Postteat and Retention Test Means and Standard Devlatlpna 

For the Three Experlmehtal Groups 



GROUP 



PRETEST 



MEAN 



SD 



POSTTEST 



MEAN 



SD 



RETENTION 



MEAN 



SD 



Ordering Theory 

Logical 

Random 



6.842 2.007 
5.737 2.513 
7.000 2.449 



i2.789 
12.158 
12.684 * 



3.207 
3.367 
2.522 



10.842 
11.42i 
10.157 



3.005. 

2.631 

2.630 



A Bartlett's test on the variances found no significant differences (p > .05). 



V 



o Table 4 ' 
Analysis of Covarlance on Objectives Passed 

On 

The Fosttest With The Pretest As Covarlate 







RESIDUAL 


DEGREES 




SOURCE 


SUH OF SQUARES 


OF FREEDOM 

* 


F-TESTS 


Full model 


* 355.102 






Covarlate 


503.793.. 


1 .53 


22.193. 


Adjusted sum 








of squares 


356.363 


2 53 


.094 
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Table 5 

Analysis of Covarlaiice On Objectives Passed 

On 

The Retention Test With The Posttest As Covarlate 



SOURCE 


RESIDUAL 
SUM OF SQUARES 


DEGREES 
OF FREEDOM 


. F-TESTS 


Full model 


167.031 






Covarlate 


411.666 


1 53 


, 27.674 


Adjusted sum 








of squares 


272.152 


2 53. 


.231 
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Table 6 

Means and Standard Deviations 
For 

Tjme To Completion For The Three Treatment Groups 





GROUP 


mean' - 




s;d. 


Ordering Theoretic 


_104,053 




13.990 


Logical 


106.263 , 




18.423 


Random 


123^211 




23.363 


TOTAL 


ill. 175 




20.543 



Table 7 
Analysis of Variance 
On 

To Completion Of Instruction in Minutes 



SOtTRCE 



DEGREES 
OF FREEDOM 



SUM OF 
SQUARES 



MEAN 
SQUARES 

L 



r-RATIO 



Between groups 
Within groups 



2 
54 



4174.375 
19457.938 



2087 .188 
360,332 



5.792** 



TOTAL 



56 



23632.313 



** p < .01 
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Higher Objective 
#16 



Queatlons 
correct 



Lover 
Objective 
#10 



Total 4 



13 



Total 
0 



13 



number 

of 

subjects 



Figure 3. 

Contingency table format used In j:alculatlng hierarchy ' 
validation. (Results derived from ordering theory 
sequence for propoaed connection between objective 16 ' 
and objective 10). 
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Table e 

Description of Validation Results 
For 

The Ordering Theory Learning Hierarchy 

L 



LOWER 



HIGHER 



NUMBER OF EXCEPTIONS 



OBSERVEO 



EXPECTED* 



1 


3 


1 


l/.Ol 


1 


9 


0 


o/.ob 


1 


12 


0 


o/.oo 


2 


7 


1 


l/.Ol 


3 


2 


6 


o/.oo 


3 


6 


0 


o/.oo 


3 


= 16 


0 


o/.oo 


4 


1 


0 


l/.OO 


5 


1 


0 


o/.oo 


6 


' 7 


2 


1/.01+ 


6 


16 


2 


i/.oi+ 


7 


13 


0 


o/.oo 


7 


14 


0 


o/.oo 


.*7 


17 


0 


o/.oo 




6 


1 


. l/.Ol 


e 


16 


2 


1/.01+ 


9 


e 


0 


l/.OO 


9 




1 


' 2/. 00 


10 


6 


0 


O/.OO 


10 


16 


0 , 


o/.oo 


11 


' IS 


'0 


l/.OO 


12 


11 


0 


1/;P0 


IS 


6 


. 1 


l/.Ol 


IS 


16 


1 


* l/.Ol 
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* The appropriate null Hypothesis Is Included In second column* 

4 

It 

+^ Connection rejected at *01 alternate hypothesis* 
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Table 9 

Description o£ Validation Results 
For 

The Logical Learning Kietarchy 



LOWER HIGHER miMBER OF EXCEPTIOMS 

OBJECTIVE OBJECTIVE OBSERVED EXPECTED* 



1 


2 


2 


1/,01+ 


2 


3 


0 


l/.OO 


3 


4 


1 


l/.Ol 


3 


5 


1 


i/.oi 


4 


6 


0 


o/.oo 


5 


7 


. 0 


o/.oo 


6 


10 


1 


i/.oi . 


7 


10 


1 


l/.Ol 


8 


10 


0 


o/.oo 


9 


10 


0 


o/.oo 


10 


J.5 


3 


1/.01+ 


11 


12 


0 


o/.oo 


12 


, 15 


5 


2/.01+ 


13 


14 


0 0 


. 2/. 00 


14 


. 15 


2 


3/. 00 


15 


16 


0 


O/.OO 


16 


17 


0 


0/,00 . 




* 







N - 19 ■ 

* The appropriate niili hypothesis Is Included in the second colinm, 
+ Connection rejected at .01 alternate hypothesis. , 
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